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Abstract 

Organizations  that  execute  contracts  must  capture  them  to  survive.  Systems  engineering  often  leads  the  technical  activities  in  the 
proposal  process.  Therefore,  it  is  important  that  systems  engineering  be  applied  effectively  on  proposals.  This  paper  introduces  a 
framework  for  developing  models  to  find  an  optimal  use  of  systems  engineering  on  proposals.  Optimization  models  developed 
using  the  framework  can  maximize  an  objective  function  of  interest  such  as  the  probability  of  contract  award  on  a  proposal  by 
leveraging  data  from  previous  proposal  efforts.  These  optimization  models  provide  recommendations  as  to  how  much  budget  to 
invest  in  systems  engineering  on  proposals  and  how  to  allocate  that  budget  across  various  systems  engineering  activities  and  to 
contributors  with  various  skill  levels.  This  paper  describes  the  framework  and  provides  guidance  for  how  to  develop  an 
optimization  model  using  the  framework  that  is  customized  for  a  particular  proposal  effort  of  interest.  The  validation  of  the 
framework  is  also  described. 
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1.  Introduction 

Organizations  that  execute  contracts  must  capture  them  to  survive.  Proposals  are  generally  how  organizations 
capture  contracts.  When  complex  systems  are  being  engineered,  systems  engineering  often  leads  the  technical 
activities  on  proposals.  Systems  engineering  can  be  used  to  help  align  the  technical  competencies  of  the  organization 
with  market  opportunities  and  to  help  define  the  solution  that  will  be  offered  in  a  proposal1.  A  relatively  new  area  of 
systems  engineering  research  is  the  use  of  systems  engineering  on  proposals2.  Little  research  has  been  done  in  this 
area,  and  a  number  of  challenges  and  opportunities  have  been  defined  related  to  the  use  of  systems  engineering  on 
proposals3.  One  of  the  identified  opportunities  that  is  especially  important  is  optimizing  the  use  of  systems  engineering 
on  proposals.  A  recently  completed  doctoral  dissertation4  focused  on  an  optimal  use  of  systems  engineering  on 
proposals.  This  conference  paper  is  a  summary  of  a  portion  of  the  doctoral  research  pertaining  to  a  framework  to  help 
decision  makers  find  an  optimal  use  of  systems  engineering  on  proposals.  For  a  more  detailed  description  of  the 
framework  or  any  aspects  of  the  framework,  the  reader  should  consult  the  dissertation. 

2.  Optimizing  the  Use  of  Systems  Engineering  on  Proposals 

Optimization  helps  decision  makers  responsible  for  proposal  success  make  the  best  decisions  that  they  can  given 
their  situation.  Optimization  is  achieved  by  minimizing  or  maximizing  a  function,  usually  subject  to  some  equality  or 
inequality  constraints5.  This  function  is  called  an  objective  function  and  mathematically  expresses  the  objective  that 
the  decision  maker  seeks  to  optimize.  Constraints  define  the  set  of  alternatives  that  are  truly  options.  A  subset  of  the 
independent  variables  in  the  objective  function  is  called  decision  variables.  Decision  variables  are  defined  as  “a 
quantity  that  a  decision  maker  controls”6.  This  section  discusses  the  objective  function,  the  constraints,  and  the 
decision  variables.  Then  a  framework  for  optimizing  the  use  of  systems  engineering  on  proposals  is  introduced. 

2.1.  The  Objective  Function 

For  proposal  management,  there  are  a  number  of  potential  objectives  that  can  be  optimized1.  These  include  proposal 
project  specific  objectives  such  as  maximizing  the  value  of  real  options,  maximizing  the  net  present  value  of  a  proposal 
opportunity,  maximizing  the  probability  of  a  contract  award,  and  minimizing  the  cost  of  preparing  a  proposal.  There 
are  also  more  broadly  focused  objectives  such  as  using  proposals  to  maximize  the  capital  position  of  an  organization, 
minimize  the  idle  rate  of  staff,  or  maximize  the  likelihood  of  repeat  business. 

For  the  framework  defined  in  this  paper,  there  are  many  potential  objectives  that  can  be  used,  but  maximizing  the 
probability  of  a  contract  award  is  the  objective  chosen  to  illustrate  the  framework.  This  objective  is  especially  relevant 
because  a  primary  purpose  of  proposal  management  is  to  be  awarded  a  contract1.  Stakeholders  of  proposal 
management,  including  project  managers,  proposal  managers,  lead  engineers,  lead  systems  engineers,  functional 
managers,  and  individual  contributors  may  benefit  from  maximizing  the  probability  of  contract  awards  for  reasons 
discussed  in  Smartt  and  Ferreira1.  Due  to  the  importance  of  this  objective,  there  are  many  examples  of  papers  that 
address  how  to  maximize  the  probability  of  winning  bids7’8'910.  Maximizing  the  probability  of  a  contract  award, 
however,  is  just  a  starting  point  for  this  type  of  analysis.  Section  6  will  discuss  potential  future  work  related  to  other 
important  objectives. 

2.2.  The  Constraints 

The  framework  presented  here  is  sufficiently  flexible  to  allow  a  number  of  different  types  of  constraints  based  upon 
the  situation.  In  general,  the  constraints  for  optimizing  the  use  of  systems  engineering  on  proposals  pertain  primarily 
to  limitations  on  available  time  or  resources.  The  discussion  in  this  paper  focuses  on  resource  limitations  and  leaves 
models  with  a  temporal  component  (e.g.,  time  constraints)  for  future  work.  In  most  cases,  organizations  submitting 
proposals  to  engineer  a  system  must  pay  for  some  or  all  of  the  proposal  costs  using  the  organization’s  resources.  In 
many  cases,  decision  makers  have  been  assigned  an  existing  budget  that  must  be  adhered  to.  In  other  cases,  there  is 
some  flexibility  about  the  budget.  Often  in  these  more  flexible  cases,  however,  rationale  has  to  be  provided  to  senior 
management  as  to  why  a  certain  amount  of  budget  is  needed.  The  budget  may  have  to  cover  a  number  of  expenses, 
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including  items  such  as  travel,  overhead  costs  (e.g.,  equipment  and  facilities)  or  the  costs  of  materials  for  generating 
the  proposal  package.  However,  when  complex  systems  are  involved,  a  leading  cost  factor  is  the  labor  costs  of  the 
time  of  employees  who  are  contributing  to  the  definition  of  the  system  being  proposed.  In  many  organizations,  labor 
costs  are  related  to  the  take  home  pay  of  the  employees.  More  senior  employees  have  higher  salaries,  and  therefore 
their  labor  costs  more  per  hour.  Less  senior  employees  have  lower  salaries,  and  therefore  their  labor  costs  less  per 
hour.  The  total  costs  of  labor  (hours  of  employee  time  multiplied  by  the  hourly  labor  rate  by  employee)  plus  the  other 
proposal  costs  must  be  less  than  or  equal  to  the  budget. 

Besides  budget,  another  potential  constraint  is  the  availability  of  personnel.  In  some  cases,  an  individual  with 
certain  specialized  skills  at  particular  levels  of  expertise  may  not  be  available  to  work  on  a  proposal  of  interest  or  may 
be  available,  but  have  a  limited  number  of  hours  that  can  be  assigned  to  the  proposal.  If  an  optimization  model 
recommends  using  an  individual  who  is  unavailable  or  using  more  labor  hours  than  the  individual  has  available,  then 
the  output  of  the  optimization  model  is  not  feasible  and  is  of  limited  use  to  the  decision  maker.  In  medium  or  large 
organizations,  there  are  often  multiple  individuals  who  have  specialized  skills  at  a  particular  level.  It  may  be  the  case 
that  one  of  the  individuals  is  not  available  but  another  one  is  available.  Therefore,  it  may  make  sense  to  specify  the 
availability  of  groups  of  individuals  with  similar  skills  and  skill  levels  versus  the  availability  of  actual  individuals.  In 
many  cases,  constraints  are  somewhat  flexible  and  models  developed  from  a  framework  such  as  this  can  be  used  to 
better  understand  the  relationships  between  the  constraint  values  and  the  objective  function  value.  This  insight  may 
help  justify  requesting  an  increased  budget  or  why  an  expert  at  a  particular  skill  should  be  added  to  the  proposal  team. 

2.3.  The  Decision  Variables 

The  decision  variables  may  vary  depending  upon  what  aspects  of  the  problem  are  under  the  control  of  the  decision 
maker  and  what  variables  are  related  to  the  objective  of  interest.  Optimizing  the  use  of  systems  engineering  on 
proposals  requires  selecting  decision  variables.  To  select  possible  decision  variables,  Smartt4  identified  a  number  of 
systems  engineering  related  factors  that  are  statistically  positively  correlated  with  the  probability  of  a  contract  award. 
These  include  the  level  of  satisfaction  of  customers  on  previous  or  ongoing  contracts,  the  relative  (to  contract  size) 
number  of  systems  engineering  labor  hours  devoted  to  key  systems  engineering  activities,  and  the  relative  (to  contract 
size)  number  of  face-to-face  contacts  between  the  supplier  and  customer  during  the  proposal  process. 

Of  the  various  types  of  factors  found  to  be  significant,  one  type  of  factor  that  is  especially  of  interest  is  the  number 
of  systems  engineering  labor  hours  devoted  to  key  systems  engineering  activities  in  the  proposal  process.  Unlike  some 
other  factors,  the  relative  systems  engineering  labor  can  be  controlled  at  the  time  of  the  proposal.  For  example, 
individuals  responsible  for  making  resource  allocation  decisions  on  proposals  can  specify  how  much  budget  is  to  be 
invested  in  the  proposal  effort  and  also  identify  how  that  budget  will  be  allocated  to  various  systems  engineering 
activities  and  employees  with  various  skill  levels.  In  contrast,  there  are  other  important  factors,  such  as  the  level  of 
customer  satisfaction  on  previous  or  ongoing  contract  work,  that  are  already  determined  prior  to  the  proposal  activities. 

Systems  engineering  labor  on  proposals  has  a  cost.  It  would  be  helpful  to  determine  an  optimal  level  of  investment 
in  systems  engineering  labor  on  a  proposal.  A  complementary  goal  is  for  any  budget  level,  to  invest  that  budget  so 
that  the  probability  of  a  contract  award  is  maximized.  It  is  desirable  to  allow  decision  makers  the  flexibility  to  define 
the  decision  variables  as  they  please. 

2.4.  An  Optimization  Framework 

This  paper  describes  a  framework  called  the  Systems  Engineering  Proposal  Optimization  Modeling  Framework 
(SEPOMF).  A  framework  is  defined  as  “a  basic  conceptual  structure  (as  of  ideas)”11.  The  SEPOMF  allows  for  various 
objective  functions,  constraints  and  decision  variables  to  be  used.  The  goal  is  to  allow  an  organization  that  is  supplying 
complex  systems  to  maximize  an  objective  of  interest  for  a  future  proposal  effort  by  leveraging  data  from  past  proposal 
efforts. 

The  SEPOMF  helps  decision  makers  develop  and  solve  optimization  problems  to  answer  important  questions.  This 
paper  explores  using  the  SEPOMF  with  a  particular  type  of  objective  function,  maximizing  the  probability  of  contract 
award,  to  determine  how  much  budget  to  invest  on  certain  systems  engineering  activities  and  what  mix  of  skill  levels 
of  employees  is  best  to  perform  those  activities.  This  paper  illustrates  the  SEPOMF  by  discussing  inputs  and  outputs 
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and  presents  the  formal  mathematical  programming  definition  of  the  optimization  problem  at  the  core  of  the  example. 
This  paper  also  discusses  how  to  apply  the  framework  to  a  particular  proposal  effort  using  historical  data  from  previous 
proposal  efforts  and  discusses  the  validation  process  for  the  framework. 

3.  Illustrating  the  SEPOMF  Using  an  Example  DSS 

The  SEPOMF  allows  analysts  to  develop  decision  support  systems  to  optimize  an  objective  of  interest  related  to 
the  use  of  systems  engineering  on  proposals.  In  order  to  define  the  SEPOMF,  it  is  necessary  to  have  a  clear  vision  of 
what  a  decision  support  system  (DSS)  generated  using  the  SEPOMF  should  do.  A  DSS  helps  people  make  decisions12. 
A  DSS  is  especially  helpful  for  problems  related  to  decision  making  that  involve  both  known  and  unknown 
components13.  A  use  case  analysis14  was  conducted  to  define  exactly  what  an  example  DSS  developed  using  the 
SEPOMF  should  do,  who  will  use  the  DSS,  and  inputs  and  outputs  of  the  DSS.  The  example  DSS  derived  from  the 
use  case  analysis  is  defined  in  this  paper,  but  has  not  yet  been  fully  developed  because  not  enough  data  is  available  to 
derive  the  parameters  values  in  an  objective  function  that  is  sufficiently  complex  to  yield  meaningful  results. 

The  inputs  to  the  example  SEPOMF  DSS  include  factors  related  to  the  contract,  the  systems  engineering 
organization  (including  the  labor  rates  and  availability  of  professionals  with  particular  systems  engineering  skill  sets 
at  particular  skill  levels),  the  competition,  the  customer  relationship,  the  proposal  effort,  and  the  budget.  The  outputs 
include  a  curve  that  expresses  the  values  of  the  objective  function  vs.  the  budget  spent.  For  the  example  DSS,  the 
curve  is  probability  of  contract  award  vs.  budget  spent.  The  example  DSS  outputs  the  number  of  labor  hours  to  invest 
in  certain  systems  engineering  activities  and  particular  skill  levels  for  those  activities. 

At  the  core  of  each  SEPOMF  DSS  is  an  optimization  problem.  The  optimization  problem  in  the  example  SEPOMF 
DSS  is  defined  below. 


Example  SEPOMF  DSS  objective  function: 


z(B)  =  max(  P(x)) 


Example  SEPOMF  DSS  constraints: 


(1) 

(2) 

(3) 


In  this  formulation,  the  vector  x  is  the  set  of  decision  variables  such  that  xy  is  the  number  of  systems  engineering 
labor  hours  for  contributors  of  skill  level  j  on  activity  i,  uy  is  the  maximum  number  of  systems  engineering  labor  hours 
that  are  available  for  skill  level  j  on  activity  i,  %  is  the  hourly  labor  rate  for  contributors  (including  overhead  loading) 
at  skill  level  j  on  activity  i,  and  B  is  the  budget  available  for  systems  engineering  labor.  Significant  care  needs  to  be 
applied  determining  B  based  upon  the  total  resources  available  for  the  proposal.  Cost  factors  potentially  not  included 
in  the  overhead  loading  such  as  capital  expenditures,  costs  for  travel,  and  costs  for  specialized  equipment  or  supplies 
also  need  to  be  considered  when  determining  B. 

The  objective  function  P  is  a  function  that  estimates  the  probability  of  a  contract  award.  In  the  example  DSS,  P 
is  developed  through  regression  analysis  (e.g.,  logistic  regression  analysis)  using  historical  proposal  data  from  the 
organization.  The  SEPOMF  is  sufficiently  flexible  that  organizations  can  customize  the  set  of  decision  variables  to 
use  the  data  that  they  already  have  collected.  The  objective  function  can  also  be  a  function  of  variables  other  than  the 
decision  variables.  Some  of  the  other  variables  may  not  be  under  the  immediate  control  of  the  decision  makers,  such 
as  the  level  of  customer  satisfaction  on  existing  contract  work.  Nonetheless,  these  factors  may  impact  the  optimal 
strategy  for  investing  in  systems  engineering  labor  for  a  particular  proposal  opportunity.  A  more  extensive  discussion 
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of  both  the  objective  function  and  constraints  is  offered  in  the  next  section. 

4.  Process  for  Applying  the  SEPOMF 

The  SEPOMF  provides  guidance  for  decision  makers  through  the  complete  optimization  modeling  process, 
including  the  following  steps:  (1)  determine  what  historical  data  is  usable,  (2)  identify  decision  variables,  (3)  derive 
and  select  an  objective  function,  (4)  determine  constraints,  (5)  define  and  solve  an  optimization  problem,  and  (6) 
interpret  and  act  on  the  results.  Figure  1  provides  an  overview  of  the  major  steps  recommended  when  applying  the 
SEPOMF.  The  following  subsections  will  discuss  each  of  these  major  steps. 


Fig.  1.  Process  for  applying  the  SEPOMF. 

4.1.  Determine  What  Historical  Data  is  Usable 

One  of  the  greatest  challenges  in  developing  a  DSS  using  the  SEPOMF  is  determining  what  set  of  historical  data 
should  be  used  in  the  regression  analysis  to  develop  the  objective  function.  Using  historical  data  that  is  different  in 
some  fundamental  way  than  the  current  proposal  should  be  avoided.  Of  course,  the  challenge  is  determining  what 
constitutes  “fundamentally  different”.  A  set  of  questions  help  decision  makers  determine  the  appropriateness  of  using 
historical  data.  For  each  historical  proposal  effort  considered,  these  questions  pertain  to  (1)  whether  the  processes  are 
basically  the  same  that  were  used  on  the  historical  proposal  as  they  are  now,  (2)  whether  the  historical  proposal  was 
for  the  same  general  type  of  effort  (e.g.,  development  of  a  system,  maintenance  of  a  system)  as  the  current  proposal, 
and  (3)  whether  the  historical  proposal  was  for  the  same  type  of  customer. 

Generally  speaking,  affirmative  answers  to  these  questions  are  necessary  but  not  sufficient  to  justify  using  a 
historical  proposal  effort  when  developing  a  DSS.  It  is  possible  that  a  recent  proposal  for  a  similar  type  of  effort  for 
a  similar  customer  may  not  be  appropriate  for  some  other  reason.  For  example,  if  the  proposal  of  interest  is  for  a  large 
contract  and  is  expected  to  employ  a  whole  team  of  systems  engineers,  it  may  not  be  wise  to  include  a  proposal  effort 
for  a  very  small  contract  that  required  a  very  limited  systems  engineering  effort  all  performed  by  one  person.  While 
the  SEPOMF  provides  a  mechanism  to  normalize  proposal  efforts  by  contract  size,  how  systems  engineering  is  used 
on  very  small  proposals  may  be  fundamentally  different  from  how  systems  engineering  is  used  on  very  large  proposals. 

4.2.  Identify  Decision  Variables 

Analysts  may  select  any  decision  variables  that  are  meaningful  in  the  context  of  their  proposal  efforts  or  that 
correspond  with  how  the  available  historical  data  is  organized.  In  general,  the  following  should  be  satisfied  by  the  set 
of  decision  variables  selected  for  a  SEPOMF  model: 

1 .  All  decision  variables  should  relate  to  well-defined  items  in  the  problem  space. 
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2.  All  decision  variables  should  be  mutually  exclusive.  For  example,  there  should  be  no  task  or  effort  that 
could  count  toward  more  than  one  of  the  decision  variables. 

3.  The  decision  variables  should  be  organized  in  a  way  that  is  consistent  with  how  the  historical  data  that  is 
used  to  calibrate  the  regression  model  is  organized.  The  various  categories  that  were  used  to  record  the 
historical  data  should  be  considered  as  a  possible  set  of  decision  variables. 

4.3.  Derive  and  Select  an  Objective  Function 

The  objective  function  may  assume  different  forms  depending  upon  what  objective  is  being  pursued.  In  the 
example  SEPOMF  DSS,  the  objective  function  form  is  a  multivariate  logistic  regression  equation.  This  form  is 
suitable  because  the  objective  function  expresses  a  probability  of  an  event  (e.g.,  a  contract  award),  and  each  historical 
data  point  has  a  binary  outcome.  In  other  words,  each  historical  data  point  either  experienced  an  event  (e.g.,  a  contract 
award)  or  did  not  experience  an  event  (e.g.,  no  contract  award).  Logistic  regression  equations  have  been  used  to  model 
a  number  of  relationships  with  similar  structure  such  as  the  probability  that  a  family  will  buy  a  new  car  as  a  function 
of  income15  and  the  probability  that  an  organization  will  submit  a  bid  for  a  particular  bid  opportunity  as  a  function  of 
a  number  of  factors16.  A  number  of  techniques  are  described  in  texts  on  regression  analysis17,18  for  selecting  the  best 
regression  model  given  a  set  of  data.  These  include  forward  selection,  stepwise  selection,  backward  deletion,  and  best 
subsets.  These  approaches  are  applicable  to  logistic  regression  as  well  as  linear  regression.  Using  these  techniques, 
a  number  of  potentially  useful  models  can  be  developed  that  express  the  probability  of  contract  award  as  a  function 
of  the  decision  variables  and  other  pertinent  factors. 

A  number  of  metrics  exist  that  can  be  used  to  compare  various  regression  models.  These  metrics  include  indices 
also  used  in  linear  regression  analysis  that  reward  high  explanatory  power  and  penalize  excessive  terms  such  as  Akaike 
Information  Criterion  (AIC)  and  Schwartz  Criterion  (SC)17.  Low  AIC  and  SC  values  are  desirable.  Another  very 
important  metric  that  is  more  specific  to  multivariate  logistic  regression  models  is  events  per  variable  (EPV)19,20. 
Maintaining  an  adequate  EPV  essentially  constrains  the  number  of  parameters  (and  hence  number  of  factors)  that  can 
be  in  a  logistic  regression  model  based  upon  the  historical  data  available. 

4.4.  Determine  Constraints 

The  constraints  described  in  Section  3  are  determined  by  the  available  resources,  and  must  be  specified  explicitly 
in  order  for  the  optimization  model  to  produce  recommendations  that  are  feasible.  For  the  example  DSS,  these 
constraints  includes  both  the  overall  budget  (Constraint  #3)  for  the  systems  engineering  activities  on  the  proposal 
effort  as  well  as  the  actual  availability  of  individuals  in  a  category  corresponding  to  a  decision  variable  (Constraint 
#1).  For  any  category  corresponding  to  a  decision  variable,  the  number  of  labor  hours  must  be  less  than  or  equal  to 
the  number  of  available  labor  hours  for  individuals  in  that  category.  Also,  the  total  labor  costs  (number  of  labor  hours 
times  cost  per  labor  hour)  cannot  exceed  the  budget.  Another  more  subtle  constraint  is  that  the  number  of  labor  hours 
for  each  decision  variable  must  be  non-negative  (Constraint  #2).  If  this  non-negative  constraint  is  not  explicitly 
specified,  the  model  may  recommend  a  negative  number  of  labor  hours  be  devoted  to  certain  individuals  with 
particular  skill  levels.  Labor  hours,  of  course,  cannot  be  negative. 

4.5.  Define  and  Solve  an  Optimization  Problem 

Once  the  objective  function  and  constraints  are  identified,  the  optimization  problem  must  be  formally  coded  and 
solved.  Given  the  potential  mathematical  complexity  of  the  objective  function  and  constraints,  it  may  be  advisable  to 
use  a  specialized  software  package  for  solving  the  optimization  problem  that  has  a  global  optimization  capability.  The 
type  of  optimization  algorithm  used  will  depend  on  the  structure  of  the  objective  function  and  nature  of  the  constraints. 
For  example,  if  the  objective  function  is  a  continuous,  nonlinear  function,  then  one  would  use  nonlinear  programming 
to  solve  the  problem. 
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4.6.  Interpret  and  Act  on  the  Results 


After  an  optimal  solution  has  been  found,  it  is  essential  to  translate  the  model  results  into  variables  that  are 
meaningful  to  decision  makers.  For  instance,  the  example  SEPOMF  DSS  will  find  an  optimal  ratio  of  the  number  of 
systems  engineering  labor  hours  to  a  metric  quantifying  the  estimated  contract  size.  Examples  of  potential  metrics 
that  could  be  used  to  quantify  contract  size  include  the  number  of  labor  hours  projected  to  be  worked  on  the  contract, 
the  number  of  expected  requirements,  and  the  estimated  total  ownership  cost.  Careful  thought  should  be  devoted  to 
selecting  a  size  metric  that  seems  appropriate  to  correspond  to  the  level  of  systems  engineering  effort  that  is  necessary 
to  support  defining  the  proposed  offering.  It  is  important  to  use  these  ratios  and  convert  the  model  results  into  units 
of  measurement  that  are  meaningful  to  decision  makers  such  as  the  number  of  labor  hours.  Once  this  translation  is 
done,  estimates  from  the  SEPOMF  DSS  should  be  compared  to  estimates  derived  from  other  means  (e.g.,  bottoms  up 
estimates,  expert  estimates)  to  verify  that  the  SEPOMF  DSS  results  make  sense. 

When  using  a  SEPOMF  DSS  to  make  decisions,  it  is  critical  to  keep  in  mind  potential  threats  to  validity.  These 
threats  to  validity  are  considered  potential  threats  because  not  all  of  them  will  apply  to  every  DSS  developed  using 
the  SEPOMF.  It  is  important  to  discuss  threats  to  validity  because  doing  so  helps  clarify  the  risks  of  applying  a 
SEPOMF  DSS  and  reduces  the  likelihood  that  a  decision  maker  will  make  an  ill-advised  decision  using  a  SEPOMF 
DSS.  Two  types  of  validity  are  considered:  internal  and  external  validity.  Findings  from  a  study  have  internal  validity 
if  effects  observed  in  the  dependent  variable  are  actually  caused  by  the  independent  variable  and  not  by  other  factors21. 
External  validity  refers  to  whether  the  causal  relationships  from  the  study  can  be  generalized  beyond  the  study 
conditions21.  It  is  necessary  to  consider  both  types  of  validity  when  applying  the  SEPOMF. 

Table  1  provides  an  overview  of  these  potential  threats.  Each  row  (after  the  title  row)  describes  a  particular 
potential  threat  to  validity.  The  first  column  describes  whether  the  potential  threat  is  an  internal  or  an  external  threat 
to  validity.  The  second  column  describes  the  potential  threat  to  validity,  and  the  third  column  describes  possible 
mitigations  to  each  potential  threat  to  validity.  These  mitigations  may  not  in  all  cases  remove  or  totally  eliminate  the 
threat  to  validity. 


Table  1 .  Potential  threats  to  validity  when  the  SEPOMF  is  applied. 

Category  Potential  Threat  Mitigations 


Internal 


External 


Relationsmps  in  objective  function  are  not  causal. 

Optimization  process  converges  to  a  suboptimal 
local  maxima. 

Optimization  model  causes  objective  function  to 
extrapolate  (be  evaluated  outside  the  domain  of  the 
historical  data  used  to  develop  objective  function). 
There  are  systematic  differences  between  proposals 
used  to  develop  the  model  and  the  proposal  being 
optimized. 


Gather  large  samples  of  data  and  explore  competing 
theories. 

Use  global  optimization  to  improve  the  likelihood  that 
global  optimal  is  found. 

Analyze  diagnostic  statistics  to  identify  instances  of 
extrapolation  (which  may  be  hidden),  and  consider  other 
approaches  if  extrapolation  cannot  be  avoided. 

Examine  the  context  of  each  historical  proposal  effort  and 
carefully  evaluate  the  applicability  of  each  historical 
proposal  effort. 


5.  Validity  of  the  Framework 

Before  applying  any  model,  it  is  first  necessary  to  assess  its  validity.  The  SEPOMF  is  not  exactly  a  model,  but  it 
is  a  framework  that  can  be  used  to  develop  and  solve  optimization  models.  Nonetheless,  there  is  value  in  validating 
the  SEPOMF.  If  an  analyst  uses  an  invalid  framework  to  develop  and  solve  optimization  problems,  there  is  limited 
hope  of  finding  valid  optimal  solutions.  Validating  a  modeling  framework,  however,  is  by  no  means  sufficient  to 
guarantee  a  valid  result  will  ultimately  be  found.  It  is  still  possible  to  use  the  framework  to  develop  invalid  models 
and  get  bad  recommendations  from  the  DSS.  In  an  ideal  situation,  one  would  develop  a  model  based  on  a  subset  of 
data  and  use  another  subset  of  sequestered  data  (data  not  used  to  develop  the  regression  model)  to  verify  that  the 
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predicted  model  outputs  match  the  response  values  in  the  sequestered  data  set.  However,  a  sufficient  quantity  of  data 
was  not  available  to  do  this,  and  hence  this  is  left  for  future  work. 

The  SEPOMF  example  DSS  concept  with  an  objective  function  of  maximizing  the  probability  of  contract  award 
was  evaluated  using  two  methods.  In  the  first  method,  the  framework  was  presented  to  a  number  of  different 
individuals  with  varied  professional  backgrounds,  and  these  individuals  provided  feedback.  These  include  individuals 
from  academia  with  expertise  in  optimization,  regression  analysis,  and  systems  engineering.  In  addition,  a  number  of 
other  experts  with  industry  experience  in  using  systems  engineering  on  proposals  reviewed  the  framework  and 
provided  feedback.  The  framework  presented  in  this  paper  was  refined  per  inputs  received  through  this  validation 
process. 

The  second  method  involved  applying  a  structured  process  with  a  formal  presentation  and  a  set  of  carefully  crafted 
questions  to  ask  validators.  No  literature  was  found  to  guide  a  structured  process  of  validating  an  optimization 
modeling  framework.  In  the  absence  of  such  guidance,  the  model  tests  defined  by  Richardson  and  Pugh22,  originally 
conceived  for  system  dynamics  models,  were  analyzed  for  applicability  to  a  modeling  framework4.  These  model  tests 
can  be  used  to  potentially  improve  a  model  by  better  aligning  it  with  reality22.  These  model  tests  examine  a  number 
of  dimensions  of  a  valid  model.  These  dimensions  address  both  the  structure  and  behavior  of  a  model.  The  tests  cover 
consistency  between  the  model  and  the  system  or  process  being  modeled,  the  suitability  of  the  model  to  answer  the 
questions  it  is  being  used  to  answer,  and  the  model’s  ability  to  provide  true  insight  that  would  otherwise  not  be 
available  to  the  decision  maker.  From  this  set  of  model  tests,  a  handful  of  questions  were  derived  that  apply  to 
optimization  modeling  frameworks.  These  questions  serve  as  the  basis  for  validation. 

Table  2  displays  the  questions  that  were  asked  of  validators  related  to  the  SEPOMF  and  how  each  of  those  questions 
map  to  model  tests  from  Richardson  and  Pugh22.  The  first  column  in  Table  2  is  the  general  activity  type,  the  second 
describes  a  model  test,  including  whether  the  test  addresses  the  model’s  structure  or  behavior,  and  the  third  column 
presents  the  derived,  SEPOMF-specific  questions.  For  some  of  the  Richardson  and  Pugh  model  tests,  there  were  no 
analogous  questions  asked.  This  was  the  case  for  the  following  structure-related  model  test:  dimensional  consistency 
and  extreme  condition  tests  in  equations.  There  were  no  questions  asked  related  to  any  of  the  behavior  aspects  of  the 
SEPOMF. 

Four  professionals  were  asked  to  validate  the  modeling  framework.  In  order  for  a  validator’s  inputs  to  be 
considered,  the  validator  had  to  meet  certain  criteria.  Each  validator  had  to: 

•  Have  participated  in  at  least  five  proposal  efforts  from  the  supplier  perspective  where  complex  systems 
were  being  engineered, 

•  Have  used  systems  engineering  on  proposals  for  complex  systems, 

•  Have  expertise  in  systems  engineering  processes, 

•  Have  an  understanding  of  optimization  modeling,  and 

•  Have  an  understanding  of  regression  models. 

All  four  individuals  affirmed  their  qualifications. 

Separate  presentations  were  given  to  each  of  the  four  professionals.  The  presentations  provided  an  overview  of  the 
SEPOMF,  including  purpose,  scope  and  example  models,  to  ensure  validators  understood  what  they  were  validating. 
In  each  presentation  there  were  opportunities  for  validators  to  ask  questions  as  the  material  was  being  presented.  Then, 
validators  were  asked  to  independently  fill  out  a  questionnaire  verifying  that  their  credentials  satisfy  the  criteria  to 
validate.  Validators  also  answered  the  pertinent  set  of  questions  from  Table  2.  The  validator  responses  relating  to  the 
SEPOMF  were  compared  side  by  side,  and  resolutions  to  the  validator  comments  were  determined.  As  a  result  of  the 
validator  feedback,  some  details  of  how  the  framework  was  described  were  refined.  Examples  of  the  refinements 
include  preconditions  for  using  the  framework  being  more  clearly  defined  and  more  explanation  given  as  to  how  the 
values  of  the  constraints  impact  the  recommended  solution. 

6.  Conclusions  and  Future  Work 

This  paper  describes  the  SEPOMF,  discusses  how  to  use  the  SEPOMF  and  presents  information  related  to  an 
example  DSS  defined  using  the  SEPOMF.  Validation  efforts  for  the  SEPOMF  are  also  described.  For  additional 
details  related  to  the  SEPOMF,  refer  to  Smartt4.  The  SEPOMF  provides  organizations  a  way  to  optimize  how  they 
use  systems  engineering  on  proposals  to  maximize  the  probability  of  contract  awards.  Considering  the  important  role 
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systems  engineering  plays  on  proposals  for  complex  systems,  optimizing  how  systems  engineering  is  used  on 
proposals  has  the  potential  to  improve  the  chance  of  survival  for  an  organization.  However,  the  framework  presented 
in  this  paper  is  just  a  beginning  to  optimizing  the  use  of  systems  engineering  on  proposals.  More  work  should  be  done 
in  this  important  area.  To  begin  with,  a  sufficient  quantity  of  historical  proposal  effort  data  needs  to  be  collected  to 
actually  develop  a  DSS  using  the  SEPOMF.  This  will  help  further  mature  the  SEPOMF  concept.  Other  technical 
issues  will  likely  become  apparent  once  real  data  is  collected  and  an  actual  DSS  is  developed  using  that  data.  In 
addition,  applying  the  SEPOMF  to  actual  project  data  may  help  attain  user  buy-in  to  the  SEPOMF  modeling  concept 
and  facilitate  additional  validation  for  models  developed  using  the  SEPOMF. 

The  future  research  on  optimizing  the  use  of  systems  engineering  on  proposals,  however,  should  be  even  broader 
in  scope.  While  focusing  on  using  systems  engineering  on  proposals  to  be  awarded  contracts  is  a  necessary  objective, 
it  is  by  no  means  sufficient.  Other  objectives  should  be  considered.  A  key  other  objective  is  to  earn  a  profit  from 
executing  those  contracts.  It  may  be  that  when  the  profit  objective  is  considered,  different  levels  of  investments  in 
systems  engineering  labor  are  recommended.  There  could  be  a  range  of  investment  levels  that  are  sufficient  to  capture 
contracts  but  insufficient  to  propose  a  solution  that  allows  the  contract  to  be  executed  profitably1 2 3 4 5 6 7 8 9.  The  framework 
may  also  be  expanded  to  allow  multiple  objectives  (that  cannot  be  consolidated  into  a  single  function)  to  be  considered 
and  to  allow  time  constraints  (e.g.,  a  deadline  that  must  be  met)  as  well  as  resource  constraints.  A  more  expansive 
model  such  considering  multiple  objectives  and  time  has  even  more  potential  to  guide  decision  makers  in  using 
systems  engineering  on  proposals.  Gaining  a  better  understanding  of  the  relationships  between  investments  in  systems 
engineering  on  a  proposal  and  desired  outcomes  will  help  organizations  profit  and  ultimately  survive. 


Table  2.  Modeling  framework  validation  questions  derived  from  Richardson  and  Pugh  model  tests. 


Activity 

Type 

Test 

Relevant  Questions 

Consistency 

Structure  - 
Boundary 
structural  adequacy 

1 . )  Are  there  important  variables  that  are  missing  in  the  SEPOMF  formulation  that  should  be 
included?  If  so,  please  explain. 

2. )  Are  there  variables  included  in  the  SEPOMF  formulation  that  are  extraneous  and  should  be 
removed?  If  so,  please  specify  which  variables  and  explain  why  you  believe  that  they  are  extraneous. 

Suitability 

Structure  - 
Face  validity 

1. )  Do  the  equations  and  inequalities  (e.g.,  the  constraints)  presented  in  the  mathematical 
programming  formulation  of  the  optimization  model  appear  to  reasonably  represent  the  variables  and 
relationships  related  to  the  use  of  systems  engineering  on  proposals?  If  not,  please  describe  issues 
that  you  see. 

2. )  Do  you  believe  that  a  reasonable  fit  exists  between  the  variables  and  structure  of  the  SEPOMF 
and  the  essential  characteristics  of  the  use  of  systems  engineering  on  proposals? 

Structure  - 
Parameter  validity 

Are  the  variables  presented  in  the  mathematical  programming  formulation  of  the  optimization  model 
recognizable  in  terms  of  the  use  of  systems  engineering  on  proposals?  If  not,  please  explain. 

Model 

Utility  and 
Effectiveness 

Structure  - 
Appropriateness  of 
structure 

Is  the  level  of  simplicity  or  complexity  and  level  of  aggregation  or  richness  of  detail  in  the  SEPOMF 
appropriate  to  allow  a  decision  maker  to  maximize  the  probability  of  contract  award  given  a  specific 
budget?  If  not,  please  explain. 
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